Cache compilation in CI with ccache - #179

CI rebuilds every object from scratch on each of the four builds a run performs, across two container images. This wraps the compilers in ccache so the compiles hit a content-addressed store instead.

Only ccache's store is cached

build/ and .sconsign.dblite stay out of it. Those are incremental state whose correctness depends on scons's dependency scanner having recorded every input, and it doesn't record all of them — DET_INIT is read straight from the environment (SConstruct:322), and the Configure results in config.h / options_cache.py aren't tracked either. Persisting that across runs is how a stale object survives a source change.

ccache re-hashes the source, every header it includes by content, the compiler binary and the full command line on every compile. So scons still rebuilds its dependency graph from a fresh checkout every run and decides what to compile from scratch; only the individual g++ -c invocations are reused. It's stricter than scons here — DET_INIT lands in CXXFLAGS, so ccache keys on it where scons doesn't.

CCACHE_SLOPPINESS is deliberately unset: include_file_mtime / include_file_ctime trade content hashing for timestamps, and time_macros would let ccache serve a stale __DATE__/__TIME__ banner for GlobalContainerArgs.cpp:347. CCACHE_COMPILERCHECK=content covers a g++ point release landing inside a base image.

Tripwires

• ccache -s -v is printed per run — a change to a widely included header that still shows near-total direct hits means the cache isn't seeing it.
• The existing harnesses already run against the produced binaries.

A cold build is gh cache delete on the two ccache-ubuntu:* entries, then re-run. On demand rather than on a timer: ccache keys on content, so a stale entry needs a hash collision, not a stale clock, and a weekly cold build spends 14 minutes of two runners to re-assert that.

Cache lifecycle

Only master saves; PR branches can't read each other's caches anyway, and letting every PR write would churn the quota and evict what PRs restore from. Master saves whenever the compiles ran, including runs a later test step failed — the entries are content-addressed, so a failing harness says nothing about whether the objects are the right objects, and a build that compiled the whole tree and then failed the LAN test shouldn't throw the tree away. !cancelled() rather than always() so a cancelled run doesn't upload during the grace period.

CCACHE_MAXSIZE=1G. A single cold build is ~130M per image; each master push touching a widely included header adds roughly another tree's worth before LRU catches up. Entries are keyed by run_id, so GitHub's 10G repo quota evicts the old ones and only the newest prefix match matters.

Enabling

CCACHE=1 in the environment, not a scons option, so it can't stick in options_cache.py. Shared helper in scons/ccache.py, used by both SConstructs. It resolves ccache via shutil.which — scons scrubs PATH for build commands, so the usual /usr/lib/ccache symlink trick would silently do nothing here. Missing ccache with CCACHE=1 set is a hard error rather than a silent uncached build.

Verified locally

With a logging pass-through shim (no ccache on the dev machine):

• CCACHE=1, no ccache on PATH → aborts with CCACHE is set but ccache was not found on PATH.
• test/SConstruct emits ccache g++ -o TestsRunner.o -c … and the object builds.
• Top level: all 37 configure probes route through the wrapper and 3112 compile-DB entries come out wrapped, so Configure is fine with the prefixed compiler.
• CCACHE unset → zero ccache entries in compile_commands.json, and config.h / options_cache.py byte-identical. Default builds untouched.

The first master run will be a cold build that populates the cache; the speedup starts after that.

Not covered

Windows. The msys2 job needs path translation between the MSYS-style CCACHE_DIR and the Windows path actions/cache tars, which I couldn't test locally. The SConstruct side already works for it if we want it as a follow-up.